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ABSTRACT 

Objectives: To identify published closed-loop 
Bayesian mixed treatment comparisons (MTCs) and to 
summarise characteristics regarding their conduct and 
reporting. 

Design: Systematic review. 
IVIetliods: We searched multiple bibliographic 
databases (January 2006-31 July 2011) for full-text, 
English language publications of Bayesian MTCs 
comparing the effectiveness or safety of >3 
interventions based on randomised controlled trials 
and having at least one closed loop. Methodological 
and reporting characteristics of MTCs were extracted in 
duplicate and summarised descriptively. 
Results: We identified 34 Bayesian MTCs spanning 13 
clinical areas. Publication of MTCs increased over the 
5-year period; with 76.5% published during or after 
2009. MTCs included a mean (±SD) of 35.9±30.1 trials 
(n=33 459±71 233 participants) and 8.5±4.3 
interventions (85.7% pharmacological). Non- 
informative and informative prior distributions were 
reported to be used in 44.1% and 8.8% of MTCs, 
respectively, with the remainder failing to specify the 
prior used. A random-effects model was used to 
analyse the networks of trials in 58.5% of MTCs, all 
using WinBUGS; however, code was infrequently 
provided (20.6%). More than two-thirds of MTCs 
(76.5%) also conducted traditional meta-analysis. 
Methods used to evaluate convergence, 
heterogeneity and inconsistency were infrequently 
reported, but from those providing detail, methods 
appeared varied. MTCs most often used a binary effect 
measure (85.3%) and ranking of interventions based 
on probability was common (61.8%), although rarely 
displayed in a figure (8.8% of MTCs). MTCs were 
published in 24 different journals with a mean impact 
factor of 9.20±8.71. While 70.8% of journals imposed 
limits on word counts and 45.8% limits on the number 
of tables/figures, online supplements/appendices were 
allowed in 79.2% of journals. Publication of 
closed-loop Bayesian MTCs is increasing in frequency, 
but details regarding their methodology are often 
poorly described. Efforts in clarifying the appropriate 
methods and reporting of Bayesian MTCs should be of 
priority. 



ARTICLE SUMMARY 



Article focus 

■ To identify published closed-loop Bayesian mixed 
treatment comparisons (MTCs) and to summarise 
characteristics regarding their conduct and 
reporting. 

Key messages 

■ We identified 34 closed-loop Bayesian MTCs 
spanning 13 clinical areas, published in 24 dif- 
ferent journals. 

■ Closed-loop Bayesian MTCs are increasing in fre- 
quency, but details regarding their methodology 
are often poorly described. Efforts in clarifying 
the appropriate methods and reporting of 
Bayesian MTCs should be of priority. 

Strengths and limitations of this study 

■ Our systematic review adds to this existing litera- 
ture by updating results and adding new infor- 

m mation as prior reviews only included literature 
^ through 2007/2008. Unlike prior publications 
our systematic review focused only on Bayesian 
MTCs of networks with at least one closed loop 

■ Unlike prior reviews, we evaluated reporting of 
additional model characteristics in depth includ- 

Iing testing for model fit, evaluation of conver- 
gence, adjustment for covariates or multiarm 
trials, the specific priors used and availability of 
the code and aggregated study-level data. 

■ An important limitation of our review is that we 
cannot say with certainty that a lack of reporting 
means a given method or analysis was not 
undertaken (ie, the testing for convergence or 
inconsistency need not be described in a paper 
for it to have been performed by the investiga- 
tors) or that the reporting of a piece of data or 
statistical code was not considered. 



INTRODUCTION 

Clinicians and decision-makers often need to 
select from multiple available interventions 
when determining the optimal treatment for 
a disease. Ideally, high-quality randomised 
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controlled trials (RCTs) that estimate the effectiveness of 
all possible interventions directly against one another 
would be available to guide decision-making/"^ 
However, interventions are commonly compared with 
placebo or non-active control in RCTs rather than 
another active intervention. When direct comparative 
trials are completed, they typically include only two 
interventions from a larger group of possible treatments. 
As such, decision-makers are faced with a lack of 
adequate direct comparative data with which to make 
their judgements. 

In the absence of head-to-head trials, indirect compar- 
isons may provide valuable information. For example, if 
two different interventions have been evaluated against 
a common comparator, the comparative effects of the 
two interventions versus each other can be estimated 
indirectly.^ ^ Even in the presence of head-to-head data, 
indirect comparisons may add value by improving preci- 
sion of treatment effect estimates. 

Methodologies exist to indirectly compare interven- 
tions, as do modes to implement such methodologies.^ 
In the simplest form, interventions that are evaluated 
against a common comparator in separate trials can be 
compared using an anchored-indirect treatment com- 
parison approach.^ As a generalisation of indirect com- 
parisons, when more than two treatments are being 
compared indirectly, and at least one pair of treatments is 
being compared directly as well as indirectly (a closed 
loop is present), both direct and indirect types of data 
can be used to estimate effects in a mixed treatment com- 
parison (MTC) meta-analysis using a Bayesian or frequen- 
tist framework. Prior research has attempted to 
categorise the use of indirect comparisons in the medical 
literature, but either did not included Bayesian MTCs or 
collected limited data on this approach.^ The Agency 
for Healthcare Research and Quality commissioned us to 
evaluate how MTCs in published systematic reviews are 
conducted and reported. We present the findings of 
our systematic review identifying closed-loop MTCs using 
a Bayesian framework and descriptively summarise their 
methodological and reporting characteristics. 

IVIETHODS 

A systematic literature search was conducted in 
MEDLINE, the Centre for Reviews and Dissemination 
Databases (including the Database of Abstracts and 
Reviews of Effects, Health Technology Assessment and 
the National Institute for Health Research Economic 
Evaluation Database), The Cochrane Library and the 
American College of Physicians Journal Club from 1 
January 2006 through 31 July 2011. The search strategy 
in online supplementary appendix SI was used. Manual 
additions were permitted based on the citations identi- 
fied by the literature search. 

Two independent investigators assessed citations for 
inclusion in a parallel manner based on a priori defined 
criteria. Specifically, we included meta-analyses that 



compared the clinical effectiveness or safety of interven- 
tions (any pharmacological (including placebo and differ- 
ent doses), behavioural or procedural interventions) 
based on RCTs, utilised a Bayesian approach to conduct 
MTC, had at least one closed loop (see online supplemen- 
tary appendix S2) and were published in full-text and in 
the English language. There has been inconsistency in 
what constitutes a MTC in the medical literature there- 
fore, for the purposes of this systematic review a MTC was 
defined as the comparison of three or more interventions 
in which direct as well as indirect evidence was used. 
Methodological publications that presented MTCs for 
illustrative purposes and cost-effectiveness analyses were 
not considered in this systematic review, nor were individ- 
ual patient data meta-analyses. 

Two reviewers independently extracted data with dis- 
agreements resolved through discussion. For each 
included closed-loop Bayesian MTC, all published material 
including the manuscript, supplements, appendices or 
external websites which the reader of the article was 
referred to for additional information were used during 
data extraction. Therefore, the extraction of data was pre- 
dicated on the reporting of the information by the authors 
within these sources. When extracting data, we recorded 
what the authors reported without ourselves judging 
whether the methods were appropriate or not. If there was 
insufficient data from all available sources, we indicated 
'not reported' for that criterion on data extraction. 

General characteristics of each MTC were extracted 
including author and funding information, if a method- 
ologist was an author, the number and type of interven- 
tion comparisons made, number of printed pages and 
use of supplement or appendix, the number of trials 
and patients in the analyses, clinical area (eg, cardiology 
and endocrinology) and the network pattern. For the 
purpose of this project, we defined a methodologist as 
an individual having an affiliation with a department of 
statistics, biostatistics, epidemiology, clinical epidemi- 
ology or public health services, as determined by author 
information and affiliations listed in the publication.^^ 
The country in which a review was conducted was deter- 
mined by the corresponding author's affiliation. 

The network pattern^ ^ was determined by 

figures presented within the identified publication. If a 
figure was not available, we determined the pattern 
based on text descriptions of included trials. 

We also extracted information regarding the method- 
ology used to conduct the closed-loop Bayesian MTC 
including the models applied (eg, fixed vs random 
effects), description of model parameters (eg, choices of 
prior distributions), methods for assessment of model fit, 
potential bias, inconsistency and heterogeneity, use of cov- 
ariate adjustment in models, whether the model accom- 
modated multiarm trials, software utilised and availability 
of code. 

Finally, we extracted data concerning the reporting of 
results including the type of endpoint (eg, binary vs con- 
tinuous), effect size and measure of variance, use of 
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other methods to report results (eg, probability of treat- 
ment being best, claims of equivalence or non- 
inferiority) and the format/presentation of results (eg, 
text, tables and figures). Characteristics of the journals 
in which included MTCs were published were collected, 
including journal name, impact factor, allowance of sup- 
plements or appendices, and limitations on word, table 
and figure counts. 

The characteristics of the closed-loop Bayesian MTCs 
and journals were summarised descriptively. Categorical 
data are presented using frequencies and continuous 
data as means±SDs. 



RESULTS 

A total of 626 citations were identified through the data- 
base searches with an additional five MTCs identified 
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through manual review (figure 1). After full text review, 
35 articles representing 34 unique closed-loop Bayesian 
MTCs were included. ^^"^^ The publication by Orme 
et af^ analysed two distinct networks of RCTs. 

The rate of publication of closed-loop Bayesian MTCs 
increased over the 5-year search period, with 26 (76.5%) 
of the MTCs published between 2009 and 2011 compared 
with only 8 (23.5%) published prior to 2009. On average, 
6.1 ±4.8 authors were listed per publication and less than 
half of publications (47.1%) included a methodologist as 
an author (table 1). The most common country from 
which authors published MTCs was the UK (35.3%), fol- 
lowed by the USA (11.8%) and Greece (11.8%). 

Funding sources for the MTCs included governmen- 
tal/foundation (29.4%), industry (26.5%) and 
unfunded (17.6%) with 23.6% not making a statement 
regarding funding source (s). Only two publications 



o 
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Records identified through 
database searching 
Medline, CCTR, CDSR, CMR, 
HTA, ACP JC (n=626) 



Records identified 
through additional 
sources (n=5) 




Records screened after 
duplicates removed (n=572) 



Records excluded (n=341 ) 
Not a systematic review (n=1 63) 
Did not meta-analyze interventions 
for clinical efficacy (n=11 1 ) 
Did not compare more than 2 arms 
(n=67) 



'5) 
lD 



T3 
0) 
T3 

O 



Full-text articles 
assessed for eligibility 
(n=231) 



Included citations (n=35) 
representing 34 unique 
Bayesian MTC 



Full-text articles excluded (n=195) 
Not a systematic review (n=9) 
Did not meta-analyze interventions 
for clinical efficacy (n=8) 
Did not conduct indirect meta- 
analysis of more than 2 arms 
(n=113) 

Methodology for indirect comparison 
other than those of interest (n=41 ) 
MTC using frequentist methods 
(n=9) 

Bayesian MTC otherthan closed 
loop network (n=13) 
Protocol or abstract (n=3) 



Figure 1 Flow diagrann of citation inclusion and exclusion. 
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Table 1 General characteristics of Bayesian mixed ^^^1 


treatment comparisons 






n/N (%) or 


Characteristic 


mean (SD) 


Number of authors 


6.1 (4.8) 


Was a methodologist an author on the 


//1"7^\ 

16/34 (47.1) 


manuscript? 




Country 




1 lO A 


4/v34 (1 1 .o) 


1 Ik' 
Ul\ 


1^/o4 (oo.o) 


uanaaa 


^/o4 (o.y) 


Brazil 


H /o/i /o n\ 

1/o4 (^.y) 


onind 




owiTzeriana 


Q/Q/l /Q Q\ 

o/o^ {o.o) 


The Netherlands 


■i /OA /O 0\ 

i/o4 (^.y) 


Italy 


o/o4 (o.o) 


Belgium 


i/o4 (^.y) 


Greece 


/I /Q/l /H H Q\ 

4/o4 (1 1 .o) 


Funding 




Industry 


y/o4 (^D.tD) 


Laovernmeni/TounQaTion 


1 U/o4 (^y.4) 


Unfunded 


din A /H "7 o\ 

d/o4 (1 7.b) 


uiner 




NOT reponeu 


o/o4 (^o.D; 


Declared affiliation 


^/v34 (o.y) 


Health Technology Assessment 


H /O /CO o\ 

1/^ (50.0) 


r rogram 




The Cochrane Collaboration 


H /O /CO 0\ 

1/^ (50.0) 


Number of printed pages 


Ib.D (ob.o) 


Supplement or appendix published 


^0/o4 (iDo.o) 


Disease state evaluated 




Dendviourdi nediin 


^lOH \ \ \ .O) 


uaroioiogy 


O/Q/l /-I "7 o\ 


inTecTious uisease 




Endocrine 


^/o4 (o.y) 


Pulmonary 


/"^inA /c o\ 

^/o4 (5.y) 


Pain 


Q/Q/l /Q Q\ 


Dermatology 


O/O/l /c o\ 

^/v34 (5.y) 


Ophthalmology 


^/o4 (o.y) 


Rheumatology 


O/Q/l /C Q\ 

^/o4 (o.y) 


Gastroenterology 


Q/Q/l /Q Q\ 

o/o4 (o.o) 


Dental 


H /O /I /O 0\ 

i/v34 (^.y) 


Oncology 


A in A /HH 0\ 

4/v34 (1 1 .o) 


oUDSiance aouse 


-1 /Q/l /O Q\ 

i/o4 (^.y) 


Number of interventions compared* 


O C //I o\ 

0.5 (4.o) 


1 ype OT intervention 




Pharmacological 


30/35 (85.7) 


Devices 


3/35 (8.6) 


Other 


1/35 (2.9) 


Device and pharmacological 


1/35 (2.9) 


Number of trials included in network* 


35.9 (30.1) 


Number of patients included in network* 


33 459 (71 233) 


*The trial by Orme ef a/ included two individual networks and they 


are considered separately for this characteristic. 



identified an organisational affiliation, one each with 
the Health Technology Assessment Program and The 
Cochrane Collaboration. The mean number of printed 
pages per publication was 16.6±36.3 (range 4-221) and 
over half published a supplement or appendix. From 



those that did not publish a supplement or appendix, 
one publication did not have the option to do so, given 
journal (or report) specifications. 

There were 13 different categories of disease 
states evaluated among included MTCs. The mean 
number of interventions included within the analyses was 
8.5±4.3, of which most were pharmacological (85.7%) in 
nature. The mean number of trials included in the MTCs 
was 35.9±30.1 and the mean number of patients included 
was 33 459±71 233 (range 594-324 168). 

The most common model used in closed4oop 
Bayesian MTCs was a random-effects model (58.5%; 
table 2). Very few analyses reported information about 
whether there was adjustment for covariates (25.6%). Of 
the 28 MTCs that included trials with three or more 
arms, 10 (35.7%) reported use of an adjustment for 
multiarm trials. Less than half of all analyses reported 
testing model fit. Of the 15 analyses that reported 
testing model fit in some manner, the most common 
method was residual deviance (40%). More than 
two-thirds of the MTCs (76.5%) also included a trad- 
itional meta-analysis. 

Closed-loop Bayesian MTCs used WinBUGS software, 
and two also specified the use of additional software 
including the BUGS XLA Wrapper and S-Plus. The statis- 
tical WinBUGS code was made available to the reader in 
only 20.6% of cases, most often in an online supple- 
ment/appendix (71.4%). Aggregated study4evel patient 
data used in the MTC was frequently made available to 
the reader and of these 21 analyses (61.8%) it was most 
commonly published within the manuscript itself 
(85.7%). Evaluation of convergence was found in 35.3% 
of analyses, most commonly using the Gelman-Rubin 
statistic (58.3%). 

Utilised priors were reported as either non-informative 
(vague or flat) or informative in 44.1% and 8.8% of ana- 
lyses, respectively. The remaining analyses (47.1%) did 
not specify the nature of the prior distributions used. It 
was also uncommon for the actual prior distribution to 
be reported for the population treatment effect (d) and 
the between-study SD of population treatment differ- 
ences across studies (a); with only 32.4% and 29.4% of 
MTCs, respectively, reporting this information. Sensitivity 
analyses based on priors were conducted in 11.8% of 
MTCs. 

Accompanying traditional meta-analyses were 
common (61.5%). The most common method used to 
assess heterogeneity was the I^ statistic (81.3%) followed 
by the Cochrane Q-statistic (43.8%). Evaluation of het- 
erogeneity within the MTC was less common, reported 
in only 32.4% of publications. Of these 11 analyses, 
(among-study variance of true effects) was used in 
54.5% of analyses followed by between-study SD (45.5%) 
and several other less frequent methods (some MTCs 
reported multiple means to test for heterogeneity and 
therefore are counted twice in the numerator). 

Inconsistency between indirect and direct estimates 
was evaluated in 24 (70.6%) studies. One review 
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reported being unable to evaluate inconsistency due to 
lack of direct data while the remaining MTCs simply did 
not comment on inconsistency. The most common 



Table 2 Continued 



Characteristic 



n/N (%) 



Table 2 Methods characteristics in Bayesian MTCs 


Characteristic 


n/N (%) 


Conducted traditional nneta-analysis 


^d/o4 (7d.o) 


Model 




Fixed effects 




Random effects 


nr\/OA /CO o\ 
^U/o4 (bo.o) 


Fixed and random effects 


//o4 [dU.O) 


NOT reponea 


o/Q/i /H v ex 


Adjustment for covariates 


Q/Q/l /OC 


Adjustment for multiple arms in MTCs 


H r\/oo /oc -7\ 

1U/^o (oo.J) 


inciuaing triais witn tnree or more arms 




iviouei TIT TesTeo 


"1 ^ /Q/1 //I /I -W 


Residual deviance 


o/H c //I n n\ 


Deviance information criterion 


O/H C /H O 0\ 


rtesiuuai aeviance ana aeviance 


Q/H c /on n\ 


information criterion 




U— U piOTS 


H /-I C /O -7\ 


Mean sum deviation 


H /-I C /O -7\ 


Method not reported 


O/H C /HO 0\ 


Code published 


"y/oA /on f5\ 


Online supplement 


oil (71 .4) 


External website 


0/-7 /oo o\ 


Aggregate study-level data published 


^1/o4 (D1 .0) 


Manuscript 


H Q/Oi /QC 7\ 

1o/^1 (ob./j 


Online supplement 


O/Oi /Q K\ 

(y.t3j 


External website 


H /OH //I Q\ 


tvaiuaTion ot convergence 


1 ^/o4 (00. 0; 


Gelman Rubin statistic 


7/H 0 /CQ Q\ 

(00. oj 


Kernel density plot 


H /H 0 /Q Q\ 


visual pioT inspecTion 


H /H 0 /Q Q\ 

1/1 ^ (o-*jj 


Observation of chain mix 


O/H 0 /H C "7\ 

tLIXii (lb./) 


Method not reported 


0 /H 0 /HO -7\ 

(Tb.7) 


Priors 




Use of non-informative 


HC/Q/l //I /I H\ 

1 iD/o4 (44.1 j 


Use of informative priors 


Q/Q/l /Q Q\ 

o/o4 (o.oj 


Not specified 


H f5/Q/l //I T H \ 
1 D/o4 (4 / .1 ) 


Prior distribution of d reported 


H H /Q/1 /QO /I \ 
1 1/o4 \OtiA) 


Prior distribution for g reported 


H n/Q/1 /OQ /I \ 

1U/o4 (^y.4j 


oensiTiviTy analysis uaseu on priors 


/l/Q/l H 1 Q\ 
^lo^^ (1 1 .0; 


Evaluation of heterogeneity in traditional 


H C^/OC^ /^5H C\ 
1 D/^D (Dl .iDj 


meTa-anaiysis 




|2 

1 


H Q/H f5 /QH Q\ 
1 0/1 D (0I .0) 


Cochrane-Q statistic 


7/16 (43.8) 


PICO statement 


1/16 (6.3) 


Plot visualisation 


2/16 (12.5) 


L'Abbe plot 


1/16 (6.3) 


Evaluation of heterogeneity in network 


11/34 (32.4) 


meta-analysis* 




Precision (x^) 


6/11 (54.5) 


Between study SD 


5/11 (45.5) 


Heterogeneity p values 


1/11 (9.1) 


Evaluation of inconsistency* 


24/34 (70.6) 


Comparison to traditional or prior 


12/24 (50.0) 


meta-analysist 






Continued 



Inconsistency/incoherence factors 
Posterior mean residual deviance 
Method not reported 
Trial sequential analysis 
Overall inconsistency (o^w) 



4/12 (33.3) 
3/12 (25.0) 
4/12 (33.3) 
1/12 (8.3) 
1/12 (8.3) 



*Studies that used multiple methods to test heterogeneity were 
counted multiple times, in the respective categories. 
tAuthors either compared results of the MTC with a traditional 
meta-analysis that they conducted concurrently or with a 
traditional meta-analysis that was previously published. 
MTC, mixed treatment comparison; PICO, patient, intervention, 
comparator, outcome. 



method used to evaluate inconsistency was comparing 
results of the MTC to those of a traditional meta-analysis 
conducted by the authors simultaneously or a previously 
published traditional meta-analysis. 

Most analyses (85.3%) reported outcomes that were 
binary (table 3). Of these 29 analyses, ORs were the 
most commonly reported effect measure (62.1%), fol- 
lowed by relative risks (17.2%) and HRs (13.8%), among 
other less frequent measures. Of the 10 (29.4%) ana- 
lyses that reported continuous outcomes, the weighted- 
mean difference was the most common effect measure 
(80%). All analyses reported variance with 95% credible 
intervals and one also reported SEs. Most analyses did 
not report if the posterior distribution was the mean or 
median value (85.3%). Presentation of results varied, 
although most analyses used multiple media including 
tables, figures and text. 

Few analyses (8.8%) presented graphical representations 
of the posterior distributions of outcomes. Rank-ordering 
of interventions based on probability statements (includ- 
ing rankograms with the probability of a treatment being 
best, second best and so on) for a given outcome was 
reported in 21 (61.8%) of the MTCs. Only one MTC 
made claims of equivalence and two made claims of non- 
inferiority, of which two defined the minimally important 
difference required to make these statements. 

Complete details of each journal in which at least one 
MTC was published can be found in tables 4 and 5. The 
34 MTCs were published in 24 different journals, with a 
mean impact factor of 9.20±8.7l. published the most 
MTCs (6 of the 34, 17.6%) followed by Current Medical 
Research and Opinion (4 of the 34, 11.8%). The majority of 
journals (70.8%) imposed word count limits and 45.8% 
imposed table/figure limitations; however, 79.2% of jour- 
nals allowed online supplements or appendices. 

DISCUSSION 

Meta-analysis has been regarded as the most highly cited 
study design in health science. However, a drawback of 
the traditional meta-analysis is its ability to compare only 
two interventions, without the ability to simultaneously 
evaluate other comparators. This is inconsistent with 
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Table 3 Outcomes and results reporting in Bayesian 
mixed treatment comparisons 


Characteristic 


n/N (%) or 
mean (SD) 


Graphical representation of 


o/o4 (o.o) 


posterior distribution 




Ranking of outcomes 


21/34 (61 .8) 


Claims of equivalence 


i/o4 (^.y) 


Claims of non-inferiority 


2/34 (o.y) 


Minimally important difference 


8/47 (1 7.0) 


Type of outcome 




Binary 


23/34 (67.6) 


Continuous 


/I/O /I /HH 0\ 

4/34 (1 1 .8) 


Binary and continuous 


O /O A /H ^ 0\ 

6/34 (1 7.6) 


Categorical non-binary 


H /O/l /O 0\ 

1/34 (2.y) 


Binary effect measure 


2y/o4 (oJd.o) 


Relative risk 


c /oo /^^~7 o\ 

o/2y (17.2) 


OR 


H O /OO //^O H \ 

18/29 (62.1) 


HR 


4/29 (13.8) 


Multiple effect measures 


2/39 (6.9) 


Continuous effect measure 


•A r\ /o A / r\r\ a \ 

1 0/34 (29.4) 


Weighted mean difference 


8/1 U (8U.U) 


Multiple 


2/10 (20.0) 


Categorical non-binary effect measure 


H /O /I /O 0\ 

1/34 (2.9) 


Relative risk 


■4 /H /A 00\ 

1/1 (100) 


Presentation of results* 




Table 


24/34 (70.6) 


Text 


32/34 (94.1) 


Figure 


21/34 (61.8 


Posterior distribution 




Mean 


1/34 (2.9) 


Median 


4/34 (1 1 .8) 


Not reported 


29/34 (85.3) 


*Studies were counted multiple times when more than one 
method was used. 



clinical practice as in many instances there are a variety 
of interventions that exist and one must decide which is 
best. The use of statistical methods (including simple 
approaches as well as MTC meta-analysis) to compare 
greater than two interventions simultaneously is on the 
rise within the peer-reviewed literature. As recent as 
2005, a search of the medical literature yielded four pub- 
lications that utilised such methods; while in 2011, the 
number increased to 57.^^ The results of our systematic 



Table 4 Aggregate journal characteristics ^^^^ 




Yes n/N (%) or 


Characteristics 


mean (SD) 


Impact factor 


9.20 (8.71) 


Supplement or appendix allowed 


19/24 (79.2) 


Online 


17/19 (89.5) 


Not specified 


2/19 (10.5) 


Word count limit 


17/24 (70.8) 


Table count limit 


11/24 (45.8) 


Figure count limit 


11/24 (45.8) 





review also suggest that indirect comparisons, specifically 
closed-loop Bayesian MTC, have become more preva- 
lent. A recent study found that a median of three studies 
(IQR 2-6) were included per meta-analysis, with close 
to 75% of meta-analyses including five or less trials. 
Our results suggest that compared to traditional 
meta-analyses, closed-loop Bayesian MTCs are larger and 
more comprehensive. Moreover, identified MTCs were 
published in a wide variety of journals covering a range 
of disease states and thus likely to reach a large reader- 
ship given their collective mean impact factor. However, 
we found a variety of reporting strategies or a lack of 
reporting of characteristics that are important to the 
conduct of closed-loop Bayesian MTC. This may be 
related to the limited guidance as to how to conduct 
and report an MTC, a topic which has been extensively 
reviewed and summarised elsewhere. 

Prior research by Donegan et aP has attempted to cat- 
egorise published indirect comparisons and evaluate 
their quality, although advanced methods including 
Bayesian (and frequentist) MTCs were not included. Of 
the 43 included comparisons, 23 used an anchored 
indirect approach while others used hypothesis testing, 
CI overlap and meta-regression methods to draw indirect 
comparisons. The authors concluded that quality of pub- 
lished indirect comparisons, in particular the assessment 
of model assumptions and the methods used to do so, 
were suboptimal. A set of quality criteria were proposed 
by the authors to be used in future indirect compari- 
sons, specifically evaluating if the method of indirect 
comparison applied was appropriate, if methods to 
assess similarity, homogeneity and consistency were 
stated and if such methods were appropriate, and details 
of overall interpretation and reporting of results. 

Song et al^^ also have systematically reviewed previously 
published indirect comparisons and, of the 88 identi- 
fied, found only 18 using 'network or Bayesian 
approaches'. Their findings are similar to that of 
Donegan and colleagues, suggesting that the main meth- 
odological problems included unclear understanding of 
assumptions, incomplete inclusion of relevant studies, 
flawed or inappropriate methods, lack of similarity 
assessment and inappropriate combination of direct and 
indirect evidence. 

Our systematic review adds to this existing literature by 
updating results and adding new information. First, the 
aforementioned prior reviews only included literature 
through 2007/2008, making ours the most up-to-date 
review available. Unlike prior publications, our system- 
atic review focused only on Bayesian MTCs of networks 
with at least one closed loop, perhaps the most common 
method utilised of late to analyse complex networks of 
RCTs. While prior publications focused on the evalu- 
ation and reporting of assumptions made within the 
models, we evaluated additional model characteristics in 
depth including testing for model fit, evaluation of con- 
vergence, adjustment for covariates or multiarm trials, 
the specific priors used and availability of the code and 
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aggregated study-level data. Despite these differences, 
however, our findings are consistent with prior research 
and with the opinion of experts regarding the chal- 
lenges and concerns around implementing and report- 
ing these more complex statistical methods/^ 
Perhaps clearer guidance as to how to conduct and 
report these types of meta-analyses will lead to a more 
optimal and consistent approach. 

While we only characterised the methods and report- 
ing of closed-loop Bayesian MTC in this report, our 
search strategy was designed to capture MTCs regardless 
of methodological approach (including frequentist 
MTC). Of note, only a handful (n=9) of frequentist 
MTCs were identified in our search, three of which spe- 
cifically reference using the methods for MTC proposed 
by Lumley and colleagues, while the others more gener- 
ically referenced mixed-model approaches. 53-60 j^^iis 
suggests that meta-analysts at present seem to favour a 
Bayesian approach to MTC, since investigators could 
have chosen to use either a Bayesian or frequentist 
method for any of the MTC identified in our search 
(given all analysed networks with at least one closed 
loop). Given the relative paucity of frequentist models, 
we do not describe the characteristics of their methods 
and reporting in this paper but they can be found 
elsewhere. 

An important limitation of our review is that we 
cannot say with certainty that a lack of reporting means 
a given method or analysis was not undertaken (ie, the 
testing for convergence or inconsistency need not be 
described in a paper for it to have been performed by 
the investigators) or that the reporting of a piece of data 
or statistical code was not considered. However, we evalu- 
ated word, table and figure limits imposed by journals in 
which these MTCs were published and our findings do 
not suggest journal space should be an obstacle to com- 
plete reporting. Another limitation is the definition used 
to describe a methodologist. While this definition has 
been used by previous researchers in a similar topic 
area,^^ to our knowledge it has not been validated 
and therefore may not accurately depict the true involve- 
ment of an individual who considered themselves a 
methodologist. 

With the growing publication of Bayesian MTCs in the 
peer-reviewed literature and the recognised challenges of 
such methods, its appropriate use and interpretation 
becomes imperative. Efforts in clarifying the appropriate 
use and reporting of Bayesian MTC should be of priority. 
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